A probabilistic framework for mutation testing in deep neural networks

Context: Mutation Testing (MT) is an important tool in traditional Software Engineering (SE) white-box testing. It aims to artificially inject faults a system evaluate test suite's capability detect them, assuming that the suite defects finding will then translate real faults. If MT has long been used SE, it only recently started gaining attention of Deep Learning (DL) community, with researchers adapting improve testability DL models and trustworthiness systems. Objective: several techniques have proposed for MT, most them neglected stochasticity inherent resulting from training phase. Even latest approaches DL, which propose tackle through statistical approach, might give inconsistent results. Indeed, as their statistic based on fixed set sampled instances, can lead different results across instances when should be consistent any instance. Methods: In this work, we Probabilistic (PMT) approach alleviates inconsistency problem allows more decision whether mutant killed or not. Results: We show PMT effectively informed mutations evaluation using three eight mutation operators previously methods. also analyze trade-off between approximation error cost our method, showing relatively small achieved manageable cost. Conclusion: Our showed limitation current practices DNN need rethink them. believe first step direction removes lack consistency executions previous methods caused by training.